Efficient empirical computation and utilization of acoustic confusability

ABSTRACT

Efficient empirical determination, computation, and use of an acoustic confusability measure comprises: (1) an empirically derived acoustic confusability measure, comprising a means for determining the acoustic confusability between any two textual phrases in a given language, where the measure of acoustic confusability is empirically derived from examples of the application of a specific speech recognition technology, where the procedure does not require access to the internal computational models of the speech recognition technology, and does not depend upon any particular internal structure or modeling technique, and where the procedure is based upon iterative improvement from an initial estimate; (2) techniques for efficient computation of empirically derived acoustic confusability measure, comprising means for efficient application of an acoustic confusability score, allowing practical application to very large-scale problems; and (3) a method for using acoustic confusability measures to make principled choices about which specific phrases to make recognizable by a speech recognition application.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a divisional application of U.S. patent applicationSer. No. 11/932,122, filed Oct. 31, 2007, which is incorporated hereinin its entirety by this reference thereto.

BACKGROUND OF THE INVENTION

Technical Field

The invention relates to speech recognition. More particularly, theinvention relates to efficient empirical determination, computation, anduse of an acoustic confusability measure.

Description of the Prior Art

In United States Patent Application Publication No. 20020032549, it isstated:

In the operation of a speech recognition system, some acousticinformation is acquired, and the system determines a word or wordsequence that corresponds to the acoustic information. The acousticinformation is generally some representation of a speech signal, such asthe variations in voltage generated by a microphone. The output of thesystem is the best guess that the system has of the text correspondingto the given utterance, according to its principles of operation.

The principles applied to determine the best guess are those ofprobability theory. Specifically, the system produces as output the mostlikely word or word sequence corresponding to the given acoustic signal.Here, “most likely” is determined relative to two probability modelsembedded in the system: an acoustic model and a language model. Thus, ifA represents the acoustic information acquired by the system, and Wrepresents a guess at the word sequence corresponding to this acousticinformation, then the system's best guess W* at the true word sequenceis given by the solution of the following equation:W*=argmax_(W) P(A|W)P(W).Here P(A|W) is a number determined by the acoustic model for the system,and P(W) is a number determined by the language model for the system. Ageneral discussion of the nature of acoustic models and language modelscan be found in “Statistical Methods for Speech Recognition,” Jelinek,The MIT Press, Cambridge, Mass. 1999, the disclosure of which isincorporated herein by reference. This general approach to speechrecognition is discussed in the paper by Bahl et al., “A MaximumLikelihood Approach to Continuous Speech Recognition,” IEEE Transactionson Pattern Analysis and Machine Intelligence, Volume PAMI-5, pp.179-190, March 1983, the disclosure of which is incorporated herein byreference.

The acoustic and language models play a central role in the operation ofa speech recognition system: the higher the quality of each model, themore accurate the recognition system. A frequently-used measure ofquality of a language model is a statistic known as the perplexity, asdiscussed in section 8.3 of Jelinek. For clarity, this statistic willhereafter be referred to as “lexical perplexity.” It is a generaloperating assumption within the field that the lower the value of thelexical perplexity, on a given fixed test corpus of words, the betterthe quality of the language model.

However, experience shows that lexical perplexity can decrease whileerrors in decoding words increase. For instance, see Clarkson et al.,“The Applicability of Adaptive Language Modeling for the Broadcast NewsTask,” Proceedings of the Fifth International Conference on SpokenLanguage Processing, Sydney, Australia, November 1998, the disclosure ofwhich is incorporated herein by reference. Thus, lexical perplexity isactually a poor indicator of language model effectiveness.

Nevertheless, lexical perplexity continues to be used as the objectivefunction for the training of language models, when such models aredetermined by varying the values of sets of adjustable parameters. Whatis needed is a better statistic for measuring the quality of languagemodels, and hence for use as the objective function during training.

United States Patent Application Publication No. 20020032549 teaches aninvention that attempts to solve these problems by:

Providing two statistics that are better than lexical perplexity fordetermining the quality of language models. These statistics, calledacoustic perplexity and the synthetic acoustic word error rate (SAWER),in turn depend upon methods for computing the acoustic confusability ofwords. Some methods and apparatuses disclosed herein substitute modelsof acoustic data in place of real acoustic data in order to determineconfusability.

In a first aspect of the invention taught in United States PatentApplication Publication No. 20020032549, two word pronunciations l(w)and l(x) are chosen from all pronunciations of all words in fixedvocabulary V of the speech recognition system. It is the confusabilityof these pronunciations that is desired. To do so, an evaluation model(also called valuation model) of l(x) is created, a synthesizer model ofl(x) is created, and a matrix is determined from the evaluation andsynthesizer models. Each of the evaluation and synthesizer models ispreferably a hidden Markov model. The synthesizer model preferablyreplaces real acoustic data. Once the matrix is determined, aconfusability calculation may be performed. This confusabilitycalculation is preferably performed by reducing an infinite series ofmultiplications and additions to a finite matrix inversion calculation.In this manner, an exact confusability calculation may be determined forthe evaluation and synthesizer models.

In additional aspects of the invention taught in United States PatentApplication Publication No. 20020032549, different methods are used todetermine certain numerical quantities, defined below, called syntheticlikelihoods. In other aspects of the invention, (i) the confusabilitymay be normalized and smoothed to better deal with very smallprobabilities and the sharpness of the distribution, and (ii) methodsare disclosed that increase the speed of performing the matrix inversionand the confusability calculation. Moreover, a method for caching andreusing computations for similar words is disclosed.

Such teachings are yet limited and subject to improvement.

SUMMARY OF THE INVENTION

There are three related elements to the invention herein:

Empirically Derived Acoustic Confusability Measures

The first element comprises a means for determining the acousticconfusability of any two textual phrases in a given language. Somespecific advantages of the means presented here are:

-   -   Empirically Derived. The measure of acoustic confusability is        empirically derived from examples of the application of a        specific speech recognition technology. Thus, the confusability        scores assigned by the measure may be expected to reflect the        actual performance of a deployed instance of the technology, in        a particular application.    -   Depends Only on Recognizer Output. The procedure described        herein does not require access to the internal computational        models of the underlying speech recognition technology, and does        not depend upon any particular internal structure or modeling        technique, such as Hidden Markov Models (HMMs). Only the output        of the speech recognition system, comprising the sequence of        decoded phonemes, is needed.    -   Iteratively Trained. The procedure described is based upon        iterative improvement from an initial estimate, and therefore        may be expected to be superior to any a priori human assignment        of phoneme confusion scores, or to a method that makes only a        single, initial estimate of phoneme confusion scores, without        iterative improvement.        Techniques for Efficient Computation of Empirically Derived        Acoustic Confusability Measures

The second element comprises computational techniques for efficientlyapplying the acoustic confusability scoring mechanism. Previousinventions have alluded to the use of acoustic confusability measures,but notably do not discuss practical aspects of applying them. In anyreal-world practical scheme, it is often required to estimate the mutualacoustic confusability of tens of thousands of distinct phrases. Withoutefficient means of computing the measure, such computations rapidlybecome impractical. In this patent, we teach means for efficientapplication of our acoustic confusability measure, allowing practicalapplication to very large-scale problems.

Method for Using Acoustic Confusability Measures

The third element comprises a method for using acoustic confusabilitymeasures, derived by whatever means (thus, not limited to the measuredisclosed here), to make principled choices about which specific phrasesto make recognizable by a speech recognition application.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 shows a basic lattice according to the invention, where thenumbers along the left are the row coordinates, the numbers along thetop are column coordinates, and the small dots are the nodes of thelattice. The coordinates are used to identify the nodes of the lattice,in the form (row coordinate, column coordinate). Thus, the coordinatesof the node in the upper-right corner are (0, 4);

FIG. 2 shows a basic lattice with actual phonemes according to theinvention, where the purely symbolic phonemes d₁ etc. have been replacedby actual phonemes from the SAMPA phoneme alphabet for US English. Thetrue phoneme sequence shown is a pronunciation of the English word“hazy,” the decoded phoneme sequence is a pronunciation of the word“raise;”

FIG. 3 shows a basic lattice with actual decoding costs according to theinvention, where the symbolic decoding costs δ(d|t) have been replacedby the starting values proposed in the text;

FIG. 4: Initial State of Bellman-Ford Algorithm. Each node now has a boxto record the minimum-path cost from the source node, at coordinates (0,0), to the node in question. The cost from the source node to itself is0, so that value has been filled. In accordance with the invention;

FIG. 5 shows two nodes with minimum path costs after labeling accordingto the invention, where the costs to reach nodes (0, 1) and (1, 0) havebeen determined and filled in, and the arcs of the minimum cost path ineach case have been marked, by rendering them with a thicker line;

FIG. 6 shows the state of the lattice after a next step of the algorithmaccording to the invention, where the cost of the minimum cost path to(1, 1), and the arc followed for that path, have both been determined.This is the first non-trivial step of the algorithm. The result isdetermined as described in the text;

FIG. 7 shows the state of the lattice after completion of the algorithmaccording to the invention, where every node has been labeled with itsminimum cost path, and the associated arcs have all been determined; andwhere some arbitrary choices, between paths of equal cost, have beenmade in selecting the minimum cost arcs;

FIG. 8 shows a confusatron output for typical homonyms according to theinvention, comprising a small portion of the list of homonyms, generatedfrom a grammar that comprised popular musical artist names, where theparenthesized text is the pronunciation that is shared by the twocolliding phrases and, where in each case, the list of colliding phrasesappears to the right, enclosed in angle brackets, with list elementsseparated by a # sign; and

FIG. 9: shows a confusatron output for typical dangerous words accordingto the invention, comprising a small portion of the list of dangerouswords, where each entry is comprised of the nominal truth, and itsclarity score, followed by a list (in order of decreasing confusability)of the other literals in the grammar that are likely to be confused withit and, where the items listed below the true literal are likelyerroneous decodings, when the given true utterance has been spoken.

DETAILED DESCRIPTION OF THE INVENTION

There are three related elements to the presently preferred embodimentof invention disclosed herein:

Empirically Derived Acoustic Confusability Measures

The first element comprises a means for determining the acousticconfusability of any two textual phrases in a given language. Somespecific advantages of the means presented here are:

-   -   Empirically Derived. The measure of acoustic confusability is        empirically derived from examples of the application of a        specific speech recognition technology. Thus, the confusability        scores assigned by the measure may be expected to reflect the        actual performance of a deployed instance of the technology, in        a particular application.    -   Depends Only on Recognizer Output. The procedure described        herein does not require access to the internal computational        models of the underlying speech recognition technology, and does        not depend upon any particular internal structure or modeling        technique, such as Hidden Markov Models (HMMs). Only the output        of the speech recognition system, comprising the sequence of        decoded phonemes, is needed.    -   Iteratively Trained. The procedure described is based upon        iterative improvement from an initial estimate, and therefore        may be expected to be superior to any a priori human assignment        of phoneme confusion scores, or to a method that makes only a        single, initial estimate of phoneme confusion scores, without        iterative improvement.        Techniques for Efficient Computation of Empirically Derived        Acoustic Confusability Measures

The second element comprises computational techniques for efficientlyapplying the acoustic confusability scoring mechanism. Previousinventions have alluded to the use of acoustic confusability measures,but notably do not discuss practical aspects of applying suchmechanisms. In any real-world practical scheme, it is often required toestimate the mutual acoustic confusability of tens of thousands ofdistinct phrases. Without efficient means of computing the measure, suchcomputations rapidly become impractical. In this patent, we teach meansfor efficient application of our acoustic confusability score, allowingpractical application to very large-scale problems.

Method for Using Acoustic Confusability Measures

The third element comprises a method for using acoustic confusabilitymeasures, derived by whatever means (thus, not limited to the measuredisclosed here), to make principled choices about which specific phrasesto make recognizable by a speech recognition application.

Empirically Derived Acoustic Confusability Measure

The immediately following discussion explains how to derive and computean empirically derived acoustic confusability measure. The discussion isdivided into several subsections;

In Section 1, we establish some notation and nomenclature, common to theinvention as a whole.

In Section 2, we explain how to empirically derive our acousticconfusability measure.

In Section 3, we explain how to use the output of the preceding sectionto compute the acoustic confusability of any two phrases.

1. Notation and Nomenclature

We first establish some notation and nomenclature. The symbol orexpression being defined appears in the left hand column; the associatedtext explains its meaning or interpretation. Italicized English words,in the associated text, give the nomenclature we use to refer to thesymbol and the concept.

-   u a single complete utterance, represented as an audio recording-   w a word sequence, phrase, or literal, represented as text. We will    use these terms interchangeably.-   X the corpus; thus a sequence of utterances u₁, u₂, . . . u_(C) and    associated transcriptions T₁, T₂, . . . , T_(C), where C is the    number of utterances in the corpus. To underscore that the corpus    contains audio data, we will sometimes refer to it as the audio    corpus or acoustic corpus.-   P the recognized corpus; the result of passing the audio corpus    through a given speech recognition system.-   Φ the phoneme alphabet of the human language in question. This is a    finite collection of the basic sound units of the language, denoted    by some textual names or symbols. For the purposes of this    discussion, we will use the Speech Assessment Methods Phonetic    Alphabet (SAMPA) for US English, as defined in Language Supplement,    OpenSpeech™ Recognizer, (US English), for English in the United    States (en-US), Second Edition, May 2004, page 33. Additional    discussion may be found in Wikipedia.-   q(w) a pronunciation or baseform (there may be several) of the    phrase w, represented as a sequence of phonemes φ₁, φ₂, . . . ,    φ_(Q), where Q is the number of phonemes in the pronunciation. Each    φ_(i) is a member of Φ.-   Q(w) the set of all pronunciations of w-   G a grammar, in the sense of an automatic speech recognition system;    thus comprising a representation of all phrases (also referred to as    word sequences or literals) that the system may recognize, nominally    with some symbolic meaning attached to each phrase-   L(G) the language of G; thus a list of all word sequences admissible    by G-   Q(L(G)) the set of all pronunciations of all word sequences    appearing in L(G); thus a list of one or more phoneme sequences for    each word sequence in L(G)-   R a recognizer or automatic speech recognition system; thus a    computer system that accepts utterances as input and returns    decodings. We will use the terms “recognizer,” “automatic speech    recognition system,” “speech recognition system” and “recognition    system” interchangeably; they mean the same thing.-   R_(G) a recognizer that is constrained to return only decodings that    correspond to phrases in the grammar G-   D a decoding; the output of a speech recognition system when    presented with an utterance. To exhibit the particular input    utterance associated to the decoding, we write D(u). For our    purposes, a decoding consists of a pair f, s, where f is the decoded    frame sequence (defined below), and s is the associated confidence    score. (Note: we may sometimes use D to denote the length of a    phoneme sequence, this will be clear from context.)-   s a confidence score; a number, determined by the recognition    system, that expresses the likelihood that the decoding returned by    the recognizer is correct. By assumption s lies in the interval [0,    1]; if not this can be arranged via a suitable scaling operation.    Written s(u) to exhibit the associated utterance u.-   T a transcription or true transcription; regular text in the human    language in question. To exhibit the particular utterance associated    to the transcription, we write T(u). (Note: we may sometimes use T    to denote the length of a phoneme sequence, this will be clear from    context.)-   f a frame of decoded speech; thus the recognizer's output for a    short segment of speech, nominally a phoneme in Φ. Written f(u) to    exhibit the associated utterance u.-   f=f₁f₂ . . . f_(F) a decoded frame sequence; the sequence of frames    associated to a particular decoding, where F is the number of frames    in the sequence. Written f(u) or f₁(u)f₂(u) . . . f_(F)(u) to    exhibit the associated utterance u.-   d=d₁d₂ . . . d_(N) a decoded phoneme sequence; the sequence of    phonemes, where N is the number of phonemes in the sequence, derived    from a particular decoded frame sequence, by the operations of    phoneme mapping and coalescing, explained below. Written d₁(u) d₂(u)    . . . d_(N)(u) to exhibit the associated utterance u.-   t=t₁t₂ . . . t_(Q) a true phoneme sequence; the sequence of    phonemes, where Q is the number of phonemes in the sequence, derived    from a true transcription T, by a means that is explained below.    Also known as a pronunciation of T. Written t(u) or t₁(u) t₂(u) . .    . t_(Q)(u) to exhibit the associated utterance u. Compare with the    decoded phoneme sequence, as defined above, and note that for one    and the same utterance u, the decoded phoneme sequence and true    phoneme sequence may and typically will differ, and may even contain    different numbers of phonemes.-   c(d|t) the integer-valued count of the phoneme pair d|t, derived as    explained below.-   δ(d|t) the decoding cost of decoding phoneme t as phoneme d. If    neither d nor t is the empty phoneme (defined below), this is also    referred to as the substitution cost of substituting d for t.-   δ_((i))(d|t) the decoding cost of decoding phoneme t as phoneme d,    at iteration i of the method. The index i is a so-called “dummy    index”; the same quantity may also be denoted using the dummy index    m as δ_((m))(d|t) to refer to the decoding cost at iteration m of    the method.-   ε the empty phoneme, sometimes called epsilon-   Φ+ε the augmented phoneme alphabet; the set Φ augmented with the    empty phoneme. Thus, Φ∪{ε}. Sometimes written Φ′.-   δ(d|ε) the insertion cost of inserting phoneme d into a decoding.-   δ(ε|t) the deletion cost of deleting phoneme t from a decoding.-   p(d|t) the probability of decoding a true phoneme t as the phoneme    d.-   p_((i))(d|t) the probability of decoding a true phoneme t as the    phoneme d, at iteration i of the method. The index i is a so-called    “dummy index”; the same quantity may also be denoted using the dummy    index m.-   Π={p(d|t)} a family of conditional probability models, where each    p(•|t) comprises a probability model, for each t in Φ+ε, over the    space Ω=Φ+ε.-   Π_((i))={p_((i))(d|t)} a family of conditional probability models,    at iteration i of the method. The index i is a so-called “dummy    index”; the same quantity may also be denoted using the dummy index    m.-   L a lattice; formally a directed acyclic graph comprising a set of    nodes N and a set of arcs or edges E⊂N×N.-   a an arc of L; formally an ordered pair of nodes    t, h    ∈E. If a=    t, h    , we say that a is an arc from t to h, and refer to t as the tail,    and h as the head, of arc a.-   A=a₁, a₂, . . . a_(K) a path (of length K) in L; formally a sequence    of arcs a₁, a₂, . . . a_(K) of L, with the property that the head of    arc a₁ is the tail of arc a_(i+1), for each i=1, . . . , K−1.-   l(a) the label of arc a; comprising the phoneme pair x|y, with x,    y∈Φ′ that is associated with the given arc in L

2. Method for Constructing an Empirically Derived Acoustic ConfusabilityMeasure

We first present an outline of the method, then present a detailedexplanation of how to apply the method.

Outline of Method

The method comprises two basic steps. The first step is corpusprocessing, in which the original corpus is passed through the automaticspeech recognition system of interest. This step is non-iterative; thatis, the corpus is processed just once by the recognition system. Thesecond step is development of a family of phoneme confusability models.This step is iterative; that is, it involves repeated passes over thecorpus, at each step delivering an improved family of confusabilitymodels.

Corpus Processing

We assume that we have at our disposal some large and representative setof utterances, in some given human language, with associated reliabletranscriptions. We refer to this as the corpus. By an utterance we meana sound recording, represented in some suitable computer-readable form.By transcription we mean a conventional textual representation of theutterance; by reliable we mean that the transcription may be regarded asaccurate. We refer to these transcriptions as the truth, or the truetranscriptions.

In this step, we pass the utterances through an automatic speechrecognition system, one utterance at a time. For each utterance, therecognition system generates a decoding, in a form called a decodedframe sequence, and a confidence score. As defined above, a frame is abrief audio segment of the input utterance.

The decoded frame sequence comprises the recognizer's best guess, foreach frame of the utterance, of the phoneme being enunciated, in thataudio frame. As defined above, a phoneme is one of a finite number ofbasic sound units of a human language.

This decoded frame sequence is then transformed, by a process that wedescribe below, into a much shorter decoded phoneme sequence. Theconfidence score is a measure, determined by the recognition system, ofthe likelihood that the given decoding is correct.

We then inspect the true transcription of the input utterance, and by aprocess that we describe below, transform the true transcription (whichis just regular text, in the language of interest) into a true phonemesequence.

Thus for each utterance we have confidence score, and a pair of phonemesequences: the decoded phoneme sequence, and the true phoneme sequence.We refer to this entire collection as the recognized corpus, and denoteit as P.

The recognized corpus constitutes the output of the corpus processingstep.

Iterative Development of Probability Model Family

From the preceding step, we have at our disposal the recognized corpusP, comprising a large number of pairs of phoneme sequences.

In this step, we iteratively develop a sequence of probability modelfamilies. That is, we repeatedly pass through the recognized corpus,analyzing each pair of phoneme sequences to collect informationregarding the confusability of any two phonemes. At the end of eachpass, we use the information just collected to generate an improvedfamily of probability models. We repeat the procedure until there is nofurther change in the family of probability models, or the changebecomes negligible.

It is important to understand that this step as a whole comprisesrepeated iterations. In the detailed discussion below, we describe asingle iteration, and the criterion for declaring the step as a wholecomplete.

The output of this step is a family of probability models, whichestimates the acoustic confusability of any two members of the augmentedphoneme alphabet Φ′. From these estimates, by another method that weexplain, we may then derive the acoustic confusability measure that weseek.

DETAILED DESCRIPTION OF THE METHOD

We now provide detailed descriptions of the steps outlined above.

Corpus Processing

Let X={<u₁, T₁>, . . . , <u_(C), T_(C)>} be the corpus, comprising Cpairs of utterances and transcriptions. For each <u, T> pair in X:

-   -   1. Recognize. Apply the recognizer R (or for a grammar-based        system, the recognizer R_(G), where G is a grammar that admits        every transcription in the corpus, plus possibly other phrases        that are desired to be recognized) to the utterance u, yielding        as output a decoded frame sequence f and a confidence score s.    -   2. Optionally Map Phonemes. This step is optional. Let f=f₁f₂ .        . . f_(F) be the decoded frame sequence. Apply a phoneme map m        to each element of the decoded frame sequence, yielding a new        decoded frame sequence f′=f′₁f′₂ . . . f′_(F), where each        f′_(j)=m(f_(j)).    -   The purpose of the phoneme map m is to reduce the effective size        of the phoneme alphabet, by collapsing minor variants within the        phoneme alphabet into a single phoneme. An example would be the        mapping of the “p closure” phoneme, often denoted pcl, to the        regular p phoneme. Another example would be splitting phoneme        pairs, known as diphones, into separate phonemes. This operation        can simplify the calculation, and avoids the problem of too        finely subdividing the available statistical evidence, which can        lead to unreliable estimates of phoneme confusability.    -   However, this operation may be skipped, or in what amounts to        the same thing, the map m may be the identity map on the phoneme        alphabet.    -   Note: it will be obvious to one skilled in the art, that by        suitable modification the map m may function to expand rather        than to reduce the phoneme alphabet, for instance by including        left and/or right phonetic context in the output phoneme. This        modification is also claimed as part of this invention.    -   3. Coalesce. Let f′=f′₁f′₂ . . . f′_(F), be the decoded frame        sequence, optionally after the application of Step 2. We now        perform the operation of coalescing identical sequential        phonemes in the decoded frame sequence, to obtain the decoded        phoneme sequence. This is done by replacing each subsequence of        identical contiguous phonemes that appear in f′ by a single        phoneme of the same type.    -   Thus if        f′=r r r r eI eI z z z z    -   is the decoded frame sequence, comprising 10 frames, the result        of coalescing f′ is the decoded phoneme sequence        d=r eI z.    -   Here and above, r, eI and z are all members of the phoneme        alphabet Φ. This phoneme sequence corresponds to the regular        English language word “raise.” Note that d has three elements,        respectively d₁=r, d₂=eI, and d₃=z.    -   We denote the coalescing operation by the letter g, and write        d=g(f′) for the action described above.    -   4. Generate Pronunciation of T. Let T be the transcription of u.        By lookup in the dictionary of the recognition system, or by use        of the system's automatic pronunciation generation system,        generate a pronunciation t for T, also written t=q(T). Thus if T        is the regular English word “hazy,” then one possibility is        t=h eI z i:    -   As above, h, eI, z, and i: are all members of the phoneme        alphabet Φ. Note that t has four elements, respectively t₁=h,        t₂=eI, t₃=z, and t₄=i:.    -   It should be noted that there may be more than one valid        pronunciation for a transcription T. There are a number of ways        of dealing with this:    -   (a) Decode the utterance u with a grammar-based recognizer        R_(G), where the grammar G restricts the recognizer to emit only        the transcription T(u). This is known as a “forced alignment,”        and is the preferred embodiment of the invention.    -   (b) Pick the most popular pronunciation, if this is known.    -   (c) Pick a pronunciation at random.    -   (d) Use all of the pronunciations, by enlarging the corpus to        contain as many repetitions of u as there are pronunciations of        T(u), and pairing each distinct pronunciation with a separate        instance of u.    -   (e) Pick the pronunciation that is closest, in the sense of        string edit distance, to the decoded phoneme sequence d.

By applying these steps sequentially to each element of the corpus X, weobtain the recognized corpus P={<ul, d(ul), t(ul), s(ul)>, . . . , <uC,d(uC), t(uC), s(uC)>}, or more succinctly P={<ul, dl, tl, sl>, . . . ,<uC, dC, tC, sC>}.

Iterative Development of Probability Model Family

We now give the algorithm for the iterative development of the requiredprobability model family, Π={p(d|t)}.

-   1. Begin with the recognized corpus P.-   2. Establish a termination condition τ. This condition typically    depends on one or more of: the number of iterations executed, the    closeness of match between the previous and current probability    family models, respectively Π_((m−1)) and Π_((m)), or some other    consideration. To exhibit this dependency explicitly, we write τ(m,    Π_((m−1)), Π_((m))).-   3. Define the family of decoding costs {δ₍₀₎(x|y)|x, y in Φ′} as    follows    δ₍₀₎(x|ε)=2 for each x in Φ    δ₍₀₎(ε|x)=3 for each x in Φ    δ₍₀₎(x|x)=0 for each x in Φ′    δ₍₀₎(x|y)=1 for each x, y in Φ, with x≠y.    Note: these settings are exemplary, and not a defining    characteristic of the algorithm. Practice has shown that the    algorithm is not very sensitive to these values, so long as    δ₍₀₎(x|x)=0, and the other quantities are greater than 0.-   4. Set the iteration count m to 0.-   5. For each x, y in Φ′, set the phoneme pair count c(x|y) to 0.-   6. For each entry <u, d, t, s> in P, perform the following (these    steps are explained in greater detail below):    -   a. Construct the lattice L=d×t.    -   b. Populate the lattice arcs with values drawn from the current        family of decoding costs, {δ_((m))(x|Y)}.    -   c. Apply the Bellman-Ford dynamic programming algorithm, or        Dijkstra's minimum cost path first algorithm, to find the        shortest path through this lattice, from the upper-left (source)        node to the lower-right (terminal) node. The minimum cost path        comprises a sequence of arcs A=a₁, a₂, . . . , a_(K), in the        lattice L, where the tail of arc a₁ is the source node, the head        of arc a_(K), is the terminal node, and the head of arc a_(i) is        the tail of arc a_(i+1), for each i=1, . . . K−1.    -   d. Traverse the minimum cost path determined in step c. Each arc        of the path is labeled with some pair x|y, where x and y are        drawn from Φ′. For each x|y arc that is traversed, increment the        phoneme pair count c(x|y) by 1.-   7. For each y in Φ′, compute c(y)=Σc(x|y), where the sum runs over    all x in Φ′.-   8. Estimate the family of probability models Π_((m))={p_((m))(x|y)}.    For each fixed y in Φ′, this is done by one of the following two    formulae:    -   a. If c(x|y) is non-zero for every x in Φ′, then set        p_((m))(x|y)=c(x|y)/c(y), for each x in Φ′.    -   b. If c(x|y) is zero for any x in Φ′, apply any desired        zero-count probability estimator, also known as a smoothing        estimator, to estimate p_((m))(x|y). A typical method is        Laplace's law of succession, which is        p_((m))(x|y)=(c(x|y)+1)/(c(y)+|Φ′|), for each x in Φ′.-   9. If m>0, test the termination condition τ(m, Π_((m−1)), Π_((m))).    If the condition is satisfied, return Π_((m)) as the desired    probability model family Π={p(d|t)} and stop.-   10. If the condition is not satisfied, define a new family of    decoding costs {δ_((m+1))(x|y)|x, y in Φ′} by δ_((m+1))(x|y)=−log    p_((m))(x|y). (The logarithm may be taken to any base greater than    1.)    -   Note that each p_((m))(x|y) satisfies 0<p_((m))(x|y)<1, and so        each δ_((m+1))(x|y)>0.-   11. Increment the iteration counter m and return to step 5 above.

We now provide the additional discussion promised above, to explain theoperations in Step 6 above.

Step 6a: Consider the entry <u, d, t, s> of P, with decoded phonemesequence d=d₁ d₂ . . . d_(N), containing N phonemes, and true phonemesequence t=t₁ t₂ . . . t_(Q) containing Q phonemes. Construct arectangular lattice of dimension (N+1) rows by (Q+1) columns, and withan arc from a node (i, j) to each of nodes (i+1, j), (i, j+1) and (i+1,j+1), when present in the lattice. (Note: “node (i, j)” refers to thenode in row i, column j of the lattice.) The phrase “when present in thelattice” means that arcs are created only for nodes with coordinatesthat actually lie within the lattice. Thus, for a node in the rightmostcolumn, with coordinates (i, Q), only the arc (i, Q) γ(i+1, Q) iscreated.)Step 6b: Label

-   -   each arc (i, j)→(i, j+1) with the cost δ_((m))(ε|t_(j))    -   each arc (i, j)→(i+1, j) with the cost δ_((m))(d_(i)|ε)    -   each arc (i, j)→(i+1, j+1) with the cost δ_((m))(d_(i)|t_(j)).

An example of such a lattice appears, in various versions, in FIGS. 1,2, and 3 below. FIG. 1 exhibits the lattice labeled with symbols, forthe case where N=3 and Q=4, with symbolic expressions for decodingcosts. FIG. 2 shows the lattice for the particular case d=r eI z and t=heI z i:, again with symbolic expressions for decoding costs. FIG. 3shows the same lattice, with the actual decoding costs for iteration 0filled in.

Step 6c: The Bellman-Ford dynamic programming algorithm is a well-knownmethod for finding the shortest path through a directed graphic with nonegative cycles. We apply it here to find the shortest path from thesource node, which we define as node (0, 0), to the terminal node, whichwe define as node (N, Q).

FIGS. 4, 5, 6, and 7 below demonstrate the application of theBellman-Ford algorithm to the example of FIG. 3.

FIG. 4 shows the initial state of the algorithm, with the source nodelabeled with the minimum cost for reaching that node from the sourcenode, which of course is 0.

FIG. 5 shows the state of the algorithm after labeling nodes (0, 1) and(1, 0) with the minimum cost for reaching those nodes from the sourcenode. The arcs traversed to yield the minimum cost has also beenexhibited, by thickening the line of the arc.

Because there is only a single arc incident on each of these nodes, theminimum costs are respectively 0+3=3 and 0+2=2. In each case, thisquantity is determined as (minimum cost to reach the immediatelypreceding node)+(cost of traversing the arc from the immediatelypreceding node).

FIG. 6 shows the state of the algorithm after labeling node (1, 1). Thecomputation here is less trivial, and we review it in detail. Node(1, 1) has three immediate predecessors, respectively (0, 0), (0, 1) and(1, 0). Each node has been labeled with its minimum cost, and so we maycompute the minimum cost to (1, 1). This of course is the minimum amongthe three possible paths to (1, 1), which are:

-   -   from (0, 0), via arc (0, 0)→(1, 1), with total cost 0+1=1    -   from (0, 1), via arc (0, 1)→(1, 1), with total cost 3+2=5    -   from (1, 0), via arc (1, 0)→(1, 1), with total cost 2+3=5.

It is evident that the path from (0, 0) is the minimum cost path, andthis is indicated in FIG. 6.

By repeated application of this process, the minimum cost path from thesource node to each node of the lattice may be determined. FIG. 7 showsthe final result.

Because the arc costs are guaranteed to be non-negative, it is evidentto one skilled in the art that the same computation may be performed, atpossibly lower computational cost, using Dijkstra's shortest path firstalgorithm. The improvement follows from the fact that only the minimumcost path from the source node to the terminal node is required, and sothe algorithm may be halted as soon as this has been determined.

The output of this step is a sequence of arcs A=a₁, a₂, . . . , a_(K),in the lattice L, known to comprise the minimum cost path from thesource node to the terminal node. We write l(a) for the phoneme pair x|ythat labels the arc a.

Step 6d: For each arc a_(i) in the minimum cost path A, labeled withphoneme pair x|y=l(a_(i)), increment the counter c(x|y) by 1.

This completes the description of the method to construct an empiricallyderived acoustic confusability measure. The means of using the result ofthis algorithm to compute the acoustic confusability of two arbitraryphrases is described below.

N-Best Variant of the Method

An important variant of the just-described method to construct anempirically derived acoustic confusability measure, which can improvethe accuracy of the resulting measure, is as follows.

It is well known to those skilled in the art that the output of arecognizer R (or R_(G), for a grammar-based recognition system), maycomprise not a single decoding D, comprising a pair f, s, but aso-called “N-best list,” comprising a ranked series of alternatedecodings, written f₁, s₁, f₂, s₂, . . . , f_(B), s_(B). In this sectionwe explain a variant of the basic method described above, called the“N-Best Variant,” which makes use of this additional information. TheN-best variant involves changes to both the corpus processing step, andthe iterative development of probability model family step, as follows.

N-Best Variant Corpus Processing

In the N-best variant of corpus processing, for each utterance u, eachentry f_(i)(u), s_(i)(u) in the N-best list is treated as a separatedecoding. All other actions, taken for a decoding of u, are thenperformed as before. The result is a larger recognized corpus P′.

N-Best Variant Iterative Development of Probability Model Family

In the N-best variant of iterative development of probability modelfamily, there are two changes. First, the input is the larger recognizedcorpus, P′, developed as described immediately above. Second, in step6d, as described above, when processing a given entry <u, d, t, s> ofP′, each count c(x|y) is incremented by the value s, which is theconfidence score of the given entry, rather than by 1.

The rest of the algorithm is unchanged.

3. Method to Compute the Empirically Derived Acoustic Confusability ofTwo Phrases

In the preceding sections we described how to determine the desiredprobability model family Π={p(d|t)}. In this section we explain how touse Π to compute the acoustic confusability of two arbitrary phrases wand v.

Specifically, we give algorithms for computing two quantities, bothrelating to acoustic confusability. The first is the raw phrase acousticconfusability r(v|w). This is a measure of the acoustic similarity ofphrases v and w. The second is the grammar-relative confusionprobability p(v|w, G). This is an estimate of the probability that agrammar-constrained recognizer R_(G) returns the phrase v as thedecoding, when the true phrase was w. Note that no reference is made toany specific pronunciation, in either quantity.

In both cases, we must come to grips with the fact that the phrases vand w may have multiple acceptable pronunciations. There are a varietyof ways of dealing with this, all of which are claimed as part of thispatent.

In the process of computing these quantities, we also give expressionsthat depend upon specific pronunciations (and from which thepronunciation-free expressions are derived). These expressions haveindependent utility, and also are claimed as part of this patent.

Computation of Raw Pronunciation Acoustic Confusability r(q(v)|q(w)) andRaw Phrase Acoustic Confusability r(v|w)

We first assume that pronunciations q(w)∈Q(w) and q(v)∈Q(v) are given,and explain the computation of the raw pronunciation acousticconfusability, r(q(v)|q(w)). Then we explain methods to determine theraw phrase acoustic confusability r(v|w).

Computation of Raw Pronunciation Acoustic Confusability

Let the probability model family Π={p(d|t)} and the pronunciations q(w)and q(v) be given. Proceed as follows to compute the raw pronunciationacoustic confusability r(q(v)|q(w)):

-   1. Define the decoding costs δ(d|t)=−log p(d|t) for each d, t∈Φ′.-   2. Construct the lattice L=q(v)×q(w), and label it with phoneme    decoding costs δ(d|t), depending upon the phonemes of q(v) and q(w).    This means performing the actions of Steps 6a and 6b, as described    above, “Iterative Development of Probability Model Family,” with the    phoneme sequences q(v) and q(w) in place of d and t respectively.-   3. Perform the actions of Step 6c above to find the minimum cost    path A=a₁, a₂, . . . , a_(K), from the source node to the terminal    node of L.-   4. Compute S, the cost of the minimum cost path A, as the sum of the    decoding costs δ(l(a)) for each arc a∈A. (Recall that l(a) is the    phoneme pair x|y that labels a.) Thus,

$S = {\sum\limits_{i = 1}^{K}{{\delta\left( {l\left( a_{i} \right)} \right)}.}}$

-   5. Compute r(q(v)|q(w))=exp(−S); this is the raw pronunciation    acoustic confusability of q(v) and q(w). Here the exponential is    computed to the same base as that used for the logarithm, in    preceding steps.

Note that equivalently

${{r\left( {q(v)} \middle| {q(w)} \right)} = {\prod\limits_{i = 1}^{K}\;{p\left( {l\left( a_{i} \right)} \right)}}},$and indeed this quantity may be computed directly from the lattice L, bysuitable modification of the steps given above.

We have described here one method of computing a measure of the acousticconfusability r(q(v)|q(w)) of two pronunciations, q(w) and q(v). In whatfollows we describe methods of manipulating this measure to obtain otheruseful expressions. It is to be noted that while the expressionsdeveloped below assume the existence of some automatic means ofquantitatively expressing the confusability of two pronunciations, theydo not depend on the exact formulation presented here, and stand asindependent inventions.

Computation of Raw Phrase Acoustic Confusability

We begin by defining r(v|q(w))=Σr(q(v)|q(w)), where the sum proceedsover all q(v)∈Q(v). This accepts any pronunciation q(v) as a decoding ofv. The raw phrase acoustic confusability r(v|w), with no reference topronunciations, may then be determined by any of the following means:

-   -   1. Worst Case, Summed. Find q(w)∈Q(w) that minimizes r(w|q(w));        call this q†(w). Thus q†(w) is the pronunciation of w that is        least likely to be correctly decoded. Set r(v|w)=r(v|q†(w)).        This is the preferred implementation.    -   2. Worst Case, Individual Pronunciations. For v≠w, set        r(v|w)=max {r(q(v)|q(w))}, where the maximum is taken over all        q(v)∈Q(v) and q(w)∈Q(w). For v=w, set r(w|w)=min {r(q(w)|q(w))},        where the minimum is taken over all q(w)∈Q(w). Since higher        values of r(q(v)|q(w)) imply greater confusability, this assigns        to r(v|w) the raw pronunciation confusability of the two most        confusable pronunciations of v and w respectively. This is the        preferred method.    -   3. Most Common. Assume the two most common pronunciations of v        and w are known, respectively q*(v) and q*(w). Set        r(v|w)=r(q*(v)|q*(w)).    -   4. Average Case. Assume that a probability distribution on Q(w)        is known, reflecting the empirical distribution, within the        general population, of various pronunciations q(w) of w. Set        r(v|w)=τp(q(w))r(v|q(w)), where the sum proceeds over all        q(w)∈Q(w).    -   5. Random. Randomly select q(v)∈Q(v) and q(w)∈Q(w), and set        r(v|w)=r(q(v)|q(w)).

Those skilled in the art will observe ways to combine these methods intoadditional hybrid variants, for instance by randomly selecting q(v), butusing the most common pronunciation q*(w), and settingr(v|w)=r(q(v)|q*(w)).

Computation of Grammar-Relative Pronunciation Confusion Probabilityp(q(v)|q(w), G) and Grammar-Relative Phrase Confusion Probability p(v|w,G)

Suppose that a recognizer is constrained to recognize phrases within agrammar G. We proceed to define expressions that estimate thegrammar-relative pronunciation confusion probability p(q(v)|q(w), G),and the grammar-relative phrase confusion probability p(v|w, G).

In what follows we write L(G) for the set of all phrases admissible bythe grammar G, and Q(L(G)) for the set of all pronunciations of all suchphrases. By assumption L(G) and Q(L(G)) are both finite.

Computation of Grammar-Relative Pronunciation Confusion Probabilityp(q(v)|q(w), G)

Let two pronunciations q(v), q(w)∈Q(L(G)) be given; exact homonyms, thatis q(v)=q(w), are to be excluded. We estimate p(q(v)|q(w), G), theprobability that an utterance corresponding to the pronunciation q(w) isdecoded by the recognizer R_(G) as q(v), as follows.

-   -   1. Compute the normalizer of q(w) relative to G, written Z(q(w),        G), as Z(q(w), G)=Σr(q(x)|q(w)), where the sum extends over all        q(x)∈Q(L(G)), excluding exact homonyms (that is, cases where        q(x)=q(w), for x≠w).    -   2. Set p(q(v)|q(w), G)=r(q(v)|q(w))/Z(q(w), G).        Note: by virtue of the definition of the normalizer, this is in        fact a probability distribution over Q(L(G)).        Computation of Grammar-Relative Phrase Confusion Probability        p(v|w, G)

Let two phrases v, w∈L(G) be given. We estimate p(v|w, G), theprobability that an utterance corresponding to any pronunciation of w isdecoded by the recognizer R_(G) as any pronunciation of v, as follows.

As above we must deal with the fact that there are in general multiplepronunciations of each phrase. We proceed in a similar manner, and beginby defining p(v|q(w), G)=Σp(q(v)|q(w), G), where the sum is taken overall q(v)∈Q(v). We may then proceed by one of the following methods:

-   -   1. Worst Case, Summed. Find q(w)∈Q(w) that minimizes p(w|q(w),        G); call this q†(w). Thus q†(w) is the pronunciation of w that        is least likely to be correctly decoded. Set p(v|w,        G)=p(v|q†(w), G). This is the preferred implementation.    -   2. Worst Case, Individual Pronunciations. For v≠w, set p′(v|w,        G)=max {p(q(v)|q(w), G)}, where the maximum is taken over all        q(v)∈Q(v) and q(w)∈Q(w). For v=w, set p′(w|w, G)=min        {p(q(w)|q(w), G)}, where the minimum is taken over all        q(w)∈Q(w). Renormalize the set of numbers {p′(x|w, G)} to obtain        a new probability distribution p(x|w, G).    -   3. Most Common. Assume the most common pronunciation of w is        known, denoted q*(w). Set p(v|w, G)=p(v|q*(w), G).    -   4. Average Case. Assume the empirical distribution p(q(w)) over        Q(w) is known. Set p(v|w, G)=Σp(q(w)) p(v|q(w), G), where the        sum is taken over all q(w)∈Q(w).    -   5. Random. For any given v, w∈L(G), randomly select q(v) and        q(w) from Q(v) and Q(w) respectively, and set p′(v|w,        G)=p(q(v)|q(w), G). Renormalize the set of numbers {p′(x|w, G)}        to obtain a new probability distribution p(x|w, G).

4. Techniques for Efficient Computation of Empirically Derived AcousticConfusability Measures

In applying measures of acoustic confusability, it is typicallynecessary to compute a very large number of grammar-relativepronunciation confusion probabilities, p(q(v)|q(w), G), which ultimatelydepend upon the quantities r(q(v)|q(w)) and Z(q(w), G). We now explainthree methods for improving the efficiency of these computations.

Partial Lattice Reuse

For a fixed q(w) in Q(L(G)), it is typically necessary to compute alarge number of raw pronunciation confusability values r(q(v)|q(w)), asq(v) takes on each or many values of Q(L(G)). In principle for each q(v)this requires the construction, labeling and minimum-cost-pathcomputation for the lattice L=q(v)×q(w), and this is prohibitivelyexpensive.

This computation can be conducted more efficiently by exploiting thefollowing observation. Consider two pronunciations q(v₁)=d₁₁, d₁₂, . . ., d_(1Q1) and q(v₂)=d₂₁, d₂₂, . . . , d_(2Q2). Suppose that they share acommon prefix; that is, for some M≦Q1, Q2 we have d_(1j)=d_(2j) for j=1,. . . , M. Then the first M rows of the labeled andminimum-cost-path-marked lattice L₁=q(v₁)×q(w) can be reused in theconstruction, labeling and minimum-cost-path computation for latticeL₂=q(v₂)×q(w)

The reuse process consists of retaining the first (M+1) rows of nodes ofthe L₁ lattice, and their associated arcs, labels and minimum-cost-pathcomputation results, and then extending this to the L₂ lattice, byadjoining nodes, and associated arcs and labels, corresponding to theremaining Q2−M phonemes of q(v₂). Thereafter, the computation of therequired minimum-cost-path costs and arcs proceeds only over thenewly-added Q2−M bottom rows of L₂.

For instance, continuing the exemplary lattice illustrated earlier,suppose q(w)=h eI z i:, and take q(v₁)=r eI z (a pronunciation of“raise”) and q(v₂)=r eI t (a pronunciation of “rate”). Then to transformL₁=q(v₁)×q(w) into L₂=q(v₂)×q(w) we first remove all the bottom row ofnodes (those with row index of 3), and all arcs incident upon them.These all correspond to the phoneme “z” in q(v₁). (However, we retainall other nodes, and all labels, values and computational results thatmark them.) Then we adjoin a new bottom row of nodes, and associatedarcs, all corresponding to the phoneme “t” in q(v₂).

Note that it is possible, for example if q(v₂)=r eI (a pronunciation of“ray”), that no additional nodes need be added, to transform L₁ into L₂.Likewise, if for example q(v₂)=r eI z @ r (a pronunciation of “razor”),it is possible that no nodes need to be removed.

This procedure may be codified as follows:

-   -   1. Fix q(w) in Q(L(G)). Construct an initial “empty” lattice L₀,        consisting of only the very top row of nodes and arcs,        corresponding to q(w).    -   2. Sort Q(L(G)) lexicographically by phoneme, yielding an        enumeration q(v₁), q(v₂), . . . .    -   3. Set the iteration counter i=1.    -   4. Find the length M of the longest common prefix of        q(v_(i−1))=d_(i−1 1), d_(i−1 2), . . . , d_(i−1 Qi−1) and        q(v_(i))=d_(i 1), d_(i 2), . . . , d_(i Qi). This is the largest        integer M such that d_(i−1j)=d_(ij) for j=1, . . . , M.    -   5. Construct lattice L_(i) from L_(i−1) as follows:    -   a. Remove the bottom Q_(i−1)−M rows of nodes (and associated        arcs, costs and labels) from L_(i−1), corresponding to phonemes        d_(i−1 M+1), . . . , d_(i−1 Qi−1) of q(v_(i−1)), forming interim        lattice L*.    -   b. Adjoin Q_(i)−M rows of nodes (and associated arcs, labeled        with costs) to the bottom of L*, corresponding to phonemes        d_(i M+1), . . . , d_(i Qi) of q(v_(i)), forming lattice        L_(i)=q(v_(i))×q(w).    -   6. Execute the Bellman-Ford or Dijkstra's shortest path first        algorithm on the newly-added portion of L_(i). Compute the value        of r(q(v_(i))|q(w)) and record the result.    -   7. Increment the iteration counter i. If additional entries of        Q(L(G)) remain, go to step 4. Otherwise stop.

It will be obvious to one skilled in the art that this same techniquemay be applied, with appropriate modifications to operate on the columnsrather than the rows of the lattice in question, by keeping q(v) fixed,and operating over an enumeration q(w₁), q(w₂), . . . of Q(L(G)) tocompute a sequence of values r(q(v)|q(w₁)), r(q(v)|q(w₂)), . . . .

Pruning

One application of acoustic confusability measures is to find phraseswithin a grammar, vocabulary or phrase list that are likely to beconfused. That is, we seek pairs of pronunciations q(v), q(w), bothdrawn from Q(L(G)), with v≠w, such that r(q(v)|q(w)), and henceultimately p(q(v)|q(w), G), is large.

In principle, this involves the computation of r(q(v)|q(w)) for some|Q(L(G))|² distinct pronunciation pairs. Because it is not uncommon forQ(L(G)) to contain as many as 100,000 members, this would entail on theorder of 10 billion acoustic confusability computations. Because of thecomplexity of the computation, this is a daunting task for even a veryfast computer.

However, it is possible to simplify this computation, as follows. If itcan be established, with a small computational effort, thatr(q(v)|q(w))<<r(q(w)|q(w)), then the expensive exact computation ofr(q(v)|q(w)) need not be attempted. In this case we declare q(v) “notconfusable” with q(w), and take r(q(v)|q(w))=0 in any furthercomputations.

We refer to such a strategy as “pruning” We now describe twocomplementary methods of pruning, respectively the method ofPronunciation Lengths, and the method of Pronunciation Sequences.

Pronunciation Lengths

Consider pronunciations q(v)=d₁, d₂, . . . , d_(D) and q(w)=t₁, t₂, . .. , t_(T). Suppose for a moment that D>>T; in other words that q(v)contains many more phonemes than q(w). Then the minimum cost paththrough the lattice L=q(v)×q(w) necessarily traverses many edges labeledwith insertion costs δ(x|e), for some x in the phoneme sequence q(v).This entails a lower bound on the minimum cost path through L, which inturn entails an upper bound on r(q(v)|q(w)).

We now explain the method in detail. Let q(v)=d₁, d₂, . . . , d_(D) andq(w)=t₁, t₂, . . . , t_(T), and let a threshold Θ be given. (The valueof Θ may be a fixed number, a function of r(q(w)|q(w)), or determined insome other way.) We proceed to compute an upper bound r†(q(v)|q(w)) onr(q(v)|q(w)).

Let us write δ_(i)=δ(d_(i)|ε) for each phoneme d_(i) of q(v), where i=1,. . . , D. Sort these costs in increasing order, obtaining a sequenceδ_(i) ₁ ≦δ_(i) ₂ ≦ . . . ≦δ_(i) _(D) .

Now, because D is the number of phonemes in q(v), even if the T phonemesof q(w) are exactly matched in the minimum cost path through thelattice, that path must still traverse at least I=D−T arcs labeled withthe insertion cost of some phoneme d of q(v). In other words, the cost Sof the minimum cost path through the lattice is bounded below by the sumof the I smallest insertion costs listed above, S†=δ_(i) ₁ +δ_(i) ₂ + .. . +δ_(i) _(I) . Because S≧S†, and by definition r(q(v)|q(w))=exp(−S),if we take r†(q(v)|q(w))=exp(−S†) we have r(q(v)|q(w))≦r†(q(v)|q(w)) asdesired.

Note: the computation of the exponential can be avoided if we take B=logΘ, and equivalently check that −B≦S†.

A similar bound may be developed for the case T>>D. For this case weconsider the phoneme deletion costs δ_(i)=δ(ε|t_(i)) for each phonemet_(i) of q(w), where i=1, . . . , T. As before, we sort these costs,obtaining the sequence δ_(i) ₁ ≦δ_(i) ₂ ≦ . . . ≦δ_(i) _(T) . LettingE=T−D, we form the sum S†=δ_(i) ₁ +δ_(i) ₂ + . . . +δ_(i) _(E) , andproceed as before.

Pronunciation Sequences

The preceding method of Pronunciation Lengths required either D>>T orT>>D, where these are the lengths of the respective pronunciationsequences. We now describe a method that may be applied, under suitableconditions, when D≈T.

For each φ in Φ, define δ_(sd) ^(min)(φ)=min{δ(x|φ)|x∈Φ″}, and defineδ_(si) ^(min)(φ)=min{δ(φ|x)|x∈φ′}. Thus 67 _(sd) ^(min)(φ) is theminimum of all costs to delete φ or substitute any other phoneme for φ,and likewise δ_(si) ^(min)(φ) is the minimum of all costs to insert φ orsubstitute φ for any other phoneme. Note that these values areindependent of any particular q(v) and q(w), and may be computed oncefor all time.

To apply the method, as above let q(v)=d₁, d₂, . . . , d_(D) andq(w)=t₁, t₂, . . . , t_(T), and let a threshold Θ be given.

For each φ in Φ, define w#(φ) and v#(φ) to be the number of times thephoneme φ appears in q(w) and q(v) respectively. Let n(φ)=w#(φ)−v#(φ).

Now form the sequence W\V=φ₁, φ₂, . . . , where for each φ in F withn(φ)>0, we insert n(φ) copies of φ into the sequence. Note that a givenφ may occur multiple times in W\V, and observe that for each instance ofφ in W\V, the minimum cost path through the lattice L=q(v)×q(w) musttraverse a substitution or deletion arc for φ.

Now compute S†=Σδ_(sd) ^(min)(φ), where the sum runs over the entries ofW\V. It follows that S, the cost of the true minimal cost path throughL, is bounded below by S†. Hence we may define r†(q(v)|q(w))=exp(−S†)and proceed as before.

A similar method applies with the sequence V\V, where we insertn(φ)=v#(φ)−w#(φ) copies of φ in the sequence, for n(φ)>0. (Note theinterchange of v and w here.) We compute S†=Σδ_(si) ^(min)(φ), where thesum runs over the entries of V\ W, and proceed as above.

Incremental Computation of Confusability in a Sequence of Grammars

Suppose have two grammars, G and G′, such that L(G) and L(G′) differfrom one another by a relatively small number of phrases, and hence sothat Q(L(G)) and Q(L(G′)) differ by only a small number ofpronunciations. Let us write Q and Q′ for these two pronunciation lists,respectively.

Suppose further that we have already computed a full set ofgrammar-relative pronunciation confusion probabilities, p(q(v)|q(w), G),for the grammar G. Then we may efficiently compute a revised setp(q(v)|q(w), G′), as follows.

First observe that the value of a raw pronunciation confusion measure,r(q(v)|q(w)), is independent of any particular grammar. While Q′ maycontain some pronunciations not in Q, for which new values r(q(v)|q(w))must be computed, most will already be known. We may therefore proceedas follows.

-   -   1. Compute any r(q(v)|q(w)), for q(v), q(w) in Q′, not already        known.    -   2. Let A=Q′\Q, that is, newly added pronunciations. Let B=Q\Q′,        that is, discarded pronunciations.    -   3. Observe now that the normalizer Z(q(w), G′)=Σr(q(x)|q(w)),        where the sum extends over q(x) in Q′, excluding exact homonyms,        may be reexpressed as Z(q(w), G′)=Z(q(w),        G)+Σ_(q(x)∈A)r(q(x)|q(w))−Σ_(q(x)∉B)r(q(x)|q (w)). Moreover, the        old normalizer Z(q(w), G′) is available as the quotient        r(q(w)|q(w))/p(q(w)|q(w), G). Thus the new normalizer Z(q(w),        G′) may be computed incrementally, at the cost of computing the        two small sums.    -   4. Finally, p(q(v)|q(w), G′) may be obtained as        r(q(v)|q(w))/Z(q(w), G′) as above.

5. Methods for Using Acoustic Confusability Measures

We now present two of the primary applications of an acousticconfusability measure.

The first of these, the “Confusatron,” is a computer program that takesas input an arbitrary grammar G, with a finite language L(G), and findsphrases in L(G) that are likely to be frequent sources of error, for thespeech recognition system. The second is a method, called maximumutility grammar augmentation, for deciding in a principled way whetheror not to add a particular phrase to a grammar.

While our discussion presumes the existence of a raw pronunciationconfusability measure r(q(v)|q(w)), and/or grammar-relativepronunciation confusion probabilities p(q(v)|q(w), G), the methodspresented in this section are independent of the particular measures andprobabilities developed in this patent, and stand as independentinventions.

The Confusatron

We now explain a computer program, which we refer to as the“Confusatron,” which automatically analyzes a given grammar G to findso-called “dangerous words.” These are actually elements of L(G) withpronunciations that are easily confusable, by a given automatic speechrecognition technology.

The value of the Confusatron is in its ability to guide a speechrecognition system designer to decide what phrases are recognized withhigh accuracy within a given application, and which are not. If a phraseidentified as likely to be poorly recognized may be discarded andreplaced by another less confusable one, in the design phase, the systemis less error-prone, and easier to use. If a phrase is likely to betroublesome, but must nevertheless be included in the system, thedesigner is at least forewarned, and may attempt to take some mitigatingaction.

We begin with a description of the Confusatron's function, and its basicmode of operation. We then describe variations; all are claimed as partof the patent.

The Confusatron generates a printed report, comprising two parts.

The first part, an example of which is exhibited in FIG. 8, lists exacthomonyms. These are distinct entries v, w in L(G), with v≠w, for whichq(v)=q(w), for some q(v)∈Q(v) and q(w)∈Q(w). That is, these are distinctliterals with identical pronunciations. Thus no speech recognizer, nomatter what its performance, is able to distinguish between v and w,when the utterance presented for recognition contains no additionalcontext, and the utterance presented for recognition matches the givenpronunciation. We say that the literals v and w “collide” on thepronunciation q(v)=q(w). Generating this homonym list does not requirean acoustic confusability measure, just a complete catalog of thepronunciation set, Q(L(G)).

However, it is the second part that is really useful. Here theConfusatron automatically identifies words with distinct pronunciationsthat are nevertheless likely to be confused. This is the “dangerousword” list, an example of which is exhibited in FIG. 9.

The Confusatron operates as follows. Let G be a grammar, with finitelanguage L(G), and finite pronunciation set Q(L(G)). Let {p(q(v)|q(w),G)} be a family of grammar-relative pronunciation confusability models,either derived from an underlying raw pronunciation confusion measurer(q(v)|q(w)) as described above, or defined by independent means.

It is useful at this point to introduce the quantity C(q(w), G), calledthe “clarity” of q(w) in G. This is a statistic of our invention, whichis defined by the formula

${C\left( {{q(w)},G} \right)} = {10\mspace{14mu}{{\log_{10}\left( \frac{p\left( {\left. {q(w)} \middle| {q(w)} \right.,G} \right)}{1 - {p\left( {\left. {q(w)} \middle| {q(w)} \right.,G} \right)}} \right)}.}}$

The unit of this statistic, defined as above, is called a “deciclar,”where “clar” is pronounced to rhyme with “car.” This turns out to be aconvenient expression, and unit, in which to measure the predictedrecognizability of a given pronunciation q(w), within a given grammar G.Note that the clarity is defined with reference to a particular grammar.If the grammar is clear from context, we do not mention it or denote itin symbols.

Note that the higher the value of p(q(w)|q(w), G), which is theestimated probability that q(w) is recognized as itself, when enunciatedby a competent speaker, the larger the value of C(q(w), G). Thus highclarity pronunciations are likely to be correctly decoded, whereas lowerclarity pronunciations are less likely to be correctly decoded. Thisforms the basic operating principle of the Confusatron, which we nowstate in detail.

-   -   1. By plotting a histogram of clarity scores of correctly        recognized and incorrectly recognized pronunciations, determine        a clarity threshold Γ. Words with pronunciations with clarity        below Γ are flagged as dangerous. Note: this step presumably        need be performed only once, for a given speech recognition        technology and acoustic confusability measure.    -   2. Let a grammar G be given. From G, by well-known techniques,        enumerate its language L(G). From L(G), by use of the        functionality of the automatic speech recognition system, or by        other well-known means such as dictionary lookup, enumerate the        pronunciation set Q(L(G)).    -   3. For each w in L(G):    -   a. Compute the clarity C(q(w), G) of each q(w) in Q(w). (For        this computation, presumably any and all of the previously        described speedup techniques may be applied to reduce the        execution time of this step.)    -   b. Set the clarity of w, written C(w, G) to the minimum of        C(q(w), G), over all q(w) in Q(w). If C(w, G)<Γ, declare w to be        dangerous, and emit w and its clarity.    -   c. In conjunction with the clarity computations of step 3a,        identify and record the phrases v for which p(v|q(w), G) attains        its highest values. Emit those phrases.    -   d. In conjunction with the clarity computations of step 3a,        identify and record any exact homonyms of q(w), and emit them        separately.

Several important variations of the basic Confusatron algorithm are nownoted.

Results for Pronunciations

First, rather than aggregating and presenting clarity results C(q(w), G)over all q(w) in Q(w), it is sometimes preferable to report them forindividual pronunciations q(w). This can be useful if it is desirable toidentify particular troublesome pronunciations.

Semantic Fusion

Second, there is often some semantic label attached to distinct phrasesv and w in a grammar, such that they are known to have the same meaning.If they also have similar pronunciations (say, they differ by thepresence of some small word, such as “a”), it is possible that the valueof p(q(v)|q(w), G) is high. This may nominally cause q(w) to have lowclarity, and thereby lead to flagging w as dangerous, when in fact thepronunciations q(v) that are confusable with q(w) have same underlyingmeaning to the speech recognition application.

It is straightforward to analyze the grammar's semantic labels, whenthey are present, and accumulate the probability mass of eachp(q(v)|q(w), G) into p(q(w)|q(w), G), in those cases when v and w havethe same meaning. This process is known as “semantic fusion,” and it isa valuable improvement on the basic Confusatron, which is also claimedin this patent.

Dangerous Word Detection Only

Suppose our task is only to decide if a given pronunciation q(w) isdangerous or not, that is if C(q(w), G)<Γ. By straightforward algebra,this can be turned into an equivalent comparison p(q(w)|q(w),G)<10^((Γ/10))/(1+10^((Γ/10))). Let us write Ψ for this transformedthreshold 10^((Γ/10))/(1+10^((Γ/10))).

Recall that p(q(w)|q(w), G)=r(q(w)|q(w))/Z(q(w), G), and that thedenominator is a monotonically growing quantity, as the defining sumproceeds over all q(v) in Q(L(G)), excluding homonyms of q(w). Now bydefinition p(q(w)|q(w), G)<Ψiff r(q(w)|q(w))/Z(q(w), G)<Ψ, that is iffZ(q(w), G)>r(q(w)|q(w))/Ψ.

Thus, we can proceed by first computing r(q(w)|q(w)), then accumulatingZ(q(w), G), which is defined as Z(q(w), G)=Σr(q(x)|q(w)), where the sumruns over all non-homonyms of q(w) in Q(L(G)), and stopping as soon asthe sum exceeds r(q(w)|q(w))/Ψ. If we arrange to accumulate into the sumthe quantities r(q(x)|q(w)) that we expect to be large, say byconcentrating on pronunciations of length close to that of q(w), thenfor dangerous words we may hope to terminate the accumulation of Z(q(w),G) without proceeding all the way through Q(L(G)).

Maximum Utility Grammar Augmentation

Suppose we are given a predetermined utility U(w) for recognizing aphrase w in a speech recognition application, a prior probability p(w)of the phrase. Then we may define the value of the phrase, within agrammar G, as V(w, G)=p(w) p(w|w, G) U(w). We may then further definethe value of a grammar as the value of all its recognizable phrases;that is, V (G)=ΣV (w, G), where the sum extends over all w in L(G).

Consider now some phrase w that is not in L(G); we are trying to decidewhether to add it to G or not. On the one hand, presumably adding thephrase has some value, in terms of enabling new functionality for agiven speech recognition application, such as permitting the search, byvoice, for a given artist or title in a content catalog.

On the other hand, adding the phrase might also have some negativeimpact, if it has pronunciations that are close to those of phrasesalready in the grammar: adding the new phrase could inducemisrecognition of the acoustically close, already-present phrases.

Let us write G+w for the grammar G with w added to it. Then a principledway to decide whether or not to add a given phrase w is to compute thegain in value ΔV(w), defined as ΔV(w)=V(G+w)−V(G).

Moreover, given a list of phrases w₁, w₂, . . . , under considerationfor addition to G, this method can be used to rank their importance, byconsidering each ΔV(w_(i)), and adding the phrases in a greedy manner.By recomputing the value gains at each stage, and stopping when thevalue gain is no longer positive, a designer can be assured of notinducing any loss in value, by adding too many new phrases.

Although the invention is described herein with reference to thepreferred embodiment, one skilled in the art will readily appreciatethat other applications may be substituted for those set forth hereinwithout departing from the spirit and scope of the present invention.Accordingly, the invention should only be limited by the Claims includedbelow.

The invention claimed is:
 1. In a computer implemented method fordetermining an empirically derived acoustic confusability measure, aniterative method for development of a probability model familyΠ={p(d|t)}, comprising: providing a recognized corpus; establishing atermination condition which depends on any of: a number of iterationsexecuted; and closeness of match between a previous and currentprobability family models; defining a family of decoding costs; settingan iteration count to 0; setting a phoneme pair count to 0; for eachentry in the recognized corpus, performing the following steps:constructing a lattice; populating lattice arcs with values drawn from acurrent family of decoding costs; applying a Bellman-Ford dynamicprogramming algorithm, or a Dijkstra's shortest path algorithm, to finda shortest path through said lattice, from a source node to a terminalnode; and traversing said determined shortest path, wherein for each arcthat is traversed, the phoneme pair count is incremented by 1; for eachtranscription, computing a confidence score which is the sum of aphoneme pair value over all transcriptions paired with an utterance;estimating said probability model family; if the iteration count exceeds0, testing said termination condition; if said termination condition issatisfied, returning a desired probability model family and stopping; ifsaid termination condition is not satisfied, defining a new family ofdecoding costs and therefrom a new probability model family; andincrementing said iteration count and repeating.
 2. The method of claim1, said step of estimating said probability model family comprisingeither of: if the confidence value is non-zero for every transcription,then setting the probability to a ratio of confidence for a phoneme pairover confidence for said utterance; and if the confidence value is zerofor any transcription, then applying a desired zero-count probabilityestimator to estimate probability.
 3. The method of claim 1, the stepsof constructing said lattice, populating said lattice arcs with valuesdrawn from a current family of decoding costs, applying saidBellman-Ford dynamic programming algorithm, or said Dijkstra's shortestpath algorithm, to find said shortest path through said lattice, andtraversing said determined shortest path, wherein for each arc that istraversed, the phoneme pair count is incremented by 1, comprising: foran entry in the recognized corpus with a decoded phoneme sequencecontaining N phonemes, and a true phoneme sequence containing Qphonemes, constructing a rectangular lattice of dimension (N+1) rows by(Q+1) columns, and with an arc from a node (i, j) to each of nodes (i+1,j), (i, j+1), and (i+1, j+1), when present in said lattice, where “node(i, j)” refers to the node in row i, column j of the lattice; labeling:each arc from node (i, j) to node (i, j+1) with the costδ_((m))(ε|t_(j)) each arc from node (i, j) to node (i+1, j) with thecost δ_((m))(d_(i)|ε) each arc from node (i, j) to node (i+1, j+1) withthe cost δ_((m))(d_(i)|t_(j)), where δ_((m)) is the associated decodingcost at the mth iteration of the algorithm, and applying theBellman-Ford dynamic programming algorithm or Dijkstra's shortest pathalgorithm to find a shortest path from the source node, which is definedas node (0, 0), to the terminal node, which is defined as node (N, Q);outputting a sequence of arcs A=a₁, a₂, . . . , a_(K), in said latticecorresponding to the aforesaid minimum cost path from the source node tothe terminal node; and for each arc a_(i) in the minimum cost path A,labeled with a phoneme pair, incrementing the associated phoneme paircount by
 1. 4. The method of claim 1, further comprising computing anempirically derived acoustic confusability of two phrases by:determining said desired probability model family Π={p(d|t)}; using Π tocompute acoustic confusability of two arbitrary phrases w and v by:computing a raw phrase acoustic confusability measure, which is ameasure of the acoustic similarity of phrases v and w; and computing agrammar-relative confusion probability measure, which is an estimate ofthe probability that a grammar-constrained recognizer returns the phrasev as a decoding, when a true phrase is w.
 5. The method of claim 4, saidstep of computing a phrase acoustic confusability measure furthercomprising: given pronunciations q(w) and q(v), computing the rawpronunciation acoustic confusability by: defining decoding costs foreach phoneme; constructing a lattice L=q(v)×q(w), and labeling it withsaid phoneme decoding costs, depending upon the phonemes of q(v) andq(w); finding a minimum cost path A=a₁, a₂, . . . , a_(K), from a sourcenode to a terminal node of L; computing a cost of said minimum cost pathA, as a sum of the decoding costs for each arc a∈A; and computing a rawpronunciation acoustic confusability measure of q(v) and q(w).
 6. Themethod of claim 4, further comprising: computing a phrase acousticconfusability measure with no reference to pronunciations by any one ofthe following: worst case; most common; average case; random; and acombination of the worst case, most common, average case, and randommethods into additional hybrid variants.
 7. The method of claim 4, saidstep of computing a grammar-relative pronunciation confusion probabilitycomprising: letting L(G) be a set of all phrases admissible by a grammarG, and letting Q(L(G)) be a set of all pronunciations of all suchphrases; letting two pronunciations q(v), q(w)∈Q(L(G)) be given;estimating a probability that an utterance corresponding to apronunciation q(w) is decoded by a recognizer R_(G) as q(v), as follows:computing a normalizer of q(w) relative to G, written Z(q(w), G), asZ(q(w),G)=Σr(q(x)|q(w)), where the sum extends over all q(x)∈Q(L(G));and setting a probability p(q(v)|q(w), G)=r(q(v)|q(w))/Z(q(w), G).